Criticisms of CEV (request for links)

Probably a good idea to unpack non-ubiquitous abbreviations at least once per post, maybe even provide links, like to this page about Coherent Extrapolated Volition as a method for choosing an AI's morals.

But yeah, sorry, 2 minutes of google-fu didn't find it, and I don't particularly care to invest more, though I probably found enough of wei dai's isolated thoughts to approximate his criticisms. Good luck!

Ben's cut-down version;

Roko's related ideas;

I've criticised CEV from several directions. It seems technically and politically infeasible to me. The CEV documents mostly read like wishlists for those seeking funding from some kind of benevolent communist government, whose leader is into science fiction.

However, I haven't collected my thoughts on the topic together in one place. I figure I should probably save my energy in this area, until there is something less out-of-date, and more concrete to consider.

I think the post is this: Hacking the CEV for Fun and Profit.

I have always been rather nervous about the concept of CEV. Particularly frightening to me are the modifiers "coherent" and "extrapolated". The explanations of these terms in this document strike me as quite incoherent and hence I am forced to extrapolate to get any meaning at all from the phrase. (The fact that the document is more than six years old and proclaims it own obsolescence in its first paragraph does not instill confidence). Of course, this posting gives me further cause for concern. It seems I may also be confused about "volition".

First, let me say why "coherent" frightens me. I wish the word were "collective" instead. It is my understanding that the point of specifying that the volition be "coherent" is that we wish to filter out the incoherent bits of mankind's volition. For example, if mankind's volition were that we not build a monolithic, super-powerful AI in the first place, then that would be an incoherent wish which should be ignored. Or, if mankind's volition did not think that the conquest of death was a high priority, that too would be incoherent and ought to be ignored. The incoherent 'philosophy' of folks below the waterline cannot be allowed to trump the volition of the more rational folk above.

The above is a caricature of 'coherence' as presented in the May 2004 document. If someone else can provide a better interpretation, that would be welcome.

Next, let me say why "extrapolated" frightens me. Extrapolation ought to frighten everyone. An AI has no business looking farther into the future than its human creators. An AI has no need to extrapolate. It has no need to look far ahead into the future. Mankind and its volition are traveling into the future along with the AI. If the AI needs to know what mankind wants 1000 years from now, it should just wait for 1000 years and then ask. It will receive a much better and well informed answer than can be achieved by extrapolating.

Once again, I may be objecting here to a caricature, straw-man interpretation of 'extrapolated'. I wish the word were 'expressed'. Can anyone provide me with a better explanation than Eliezer's (2004) as to why the word 'extrapolated' is the appropriate one?

Also, does anyone have a link to the Wei Dai comments regarding "volition"? The OP's hints make this word just as mysterious and frightening to me as the other two.

I think the post is this: Hacking the CEV for Fun and Profit.

I think the link is this: Hacking the CEV for Fun and Profit.

Hacking the CEV for Fun and Profit

http://lesswrong.com/lw/2b7/hacking_the_cev_for_fun_and_profit/